The objective of this paper is visual-only self-supervised video representation learning. We make the following contributions: (i) we investigate the benefit of adding semantic-class positives to instance-based Info Noise Contrastive Estimation (InfoNCE) training, showing that this form of supervised contrastive learning leads to a clear improvement in performance; (ii) we propose a novel self-supervised co-training scheme to improve the popular infoNCE loss, exploiting the complementary information from different views, RGB streams and optical flow, of the same data source by using one view to obtain positive class samples for the other; (iii) we thoroughly evaluate the quality of the learnt representation on two different downstream tasks: action recognition and video retrieval. In both cases, the proposed approach demonstrates state-of-the-art or comparable performance with other self-supervised approaches, whilst being significantly more efficient to train, i.e. requiring far less training data to achieve similar performance.

We use the S3D architecture for all experiments. CoCLR), S3D is followed by a non-linear projection head. The projection head is removed when evaluating downstream tasks. The detailed dimensions are shown in Table 1.Stage Detail Output size: T HW C S3D followed by average pooling 1 1 When evaluating the pretrained representation for action classification, we replace the non-linear projection head with a single linear layer for the classification tasks. The history queue is used in all pretraining experiments (including both InfoNCE and CoCLR).

Weaknesses: # Related Work I believe that the related work section has been poorly executed. It simply lists numerous papers from 4 domains of existing research. This provides limited information about the position of the proposed work w.r.t these existing works. A more detailed discussion of the contrast between the proposed work and existing literature, or the similarities, or the parts that have been directly adopted is generally expected from the related work section. Furthermore, the authors may have missed the following highly relevant papers which are also discussed in my Baselines section below: 1) Yan, Xueting, et al. "ClusterFit: Improving Generalization of Visual Representations."

The objective of this paper is visual-only self-supervised video representation learning. We make the following contributions: (i) we investigate the benefit of adding semantic-class positives to instance-based Info Noise Contrastive Estimation (InfoNCE) training, showing that this form of supervised contrastive learning leads to a clear improvement in performance; (ii) we propose a novel self-supervised co-training scheme to improve the popular infoNCE loss, exploiting the complementary information from different views, RGB streams and optical flow, of the same data source by using one view to obtain positive class samples for the other; (iii) we thoroughly evaluate the quality of the learnt representation on two different downstream tasks: action recognition and video retrieval. In both cases, the proposed approach demonstrates state-of-the-art or comparable performance with other self-supervised approaches, whilst being significantly more efficient to train, i.e. requiring far less training data to achieve similar performance.